Methods and systems for dynamic monitoring through graphical user interfaces

ABSTRACT

Methods and systems for monitoring through graphical user interfaces are disclosed. In one aspect, a system is disclosed that includes a processor and data storage including instructions that, when executed by the processor, cause the system to perform operations. The operations include maintaining an input file including predetermined criteria for a plurality of factors, receiving a model dataset generated using a model, based on the input file and the model dataset, generating a first graphical user interface that includes a graphical illustration of a subset of the plurality of factors, a model risk for the model, and a selectable feature associated with a selected factor in the subset and a selected time period. The operations further include receiving through the first graphical user interface a selection of the selectable feature, and, in response, generating a second graphical user interface that recolors a portion of the first graphical user interface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/228,021, filed Apr. 12, 2021, which is a continuation of U.S. patentapplication Ser. No. 16/781,255 filed Feb. 4, 2020, which is acontinuation of U.S. patent application Ser. No. 15/963,699, filed Apr.26, 2018, which claims priority under 35 U.S.C. § 119 to U.S.Provisional Patent Application No. 62/635,224, filed Feb. 26, 2018. Theaforementioned applications are incorporated herein by reference intheir entirety.

BACKGROUND

In some cases, models may be used to model one or more factors. Forexample, a model may be created using software that generates a modeledvalue for each factor. The collection of modeled values may be referredto as a model dataset.

Because models are imperfect, a modeled value for a factor may in someinstances deviate from a value for that factor in reality. As a result,the model may similarly deviate from reality. Such deviation of themodel may be referred to as “model drift.” Model drift may pose a dangerthat the model will not accurately model reality. Danger associated withmodel drift may be referred to as “model risk.” The extent to which asingle factor and/or its associated modeled value contributes to themodel risk may be referred to as a “factor risk.”

In some instances, it may be desirable to dynamically isolate, identify,and illustrate factors that contribute to model drift and/or model risk.Existing technologies, however, which rely on static graphical userinterfaces, do not provide such dynamic isolation, identification, andillustration of factors that contribute to model drift and/or modelrisk.

SUMMARY

The disclosed embodiments describe systems and methods for dynamicmonitoring models through graphical user interfaces.

In one aspect, a system is disclosed that includes a processor and datastorage including instructions that, when executed by the processor,cause the system to perform operations. The operations includemaintaining an input file comprising predetermined criteria for aplurality of factors; receiving a model dataset generated using a model,the model dataset comprising modeled values for the factors; generating,based on the input file and the model dataset, a first graphical userinterface comprising a graphical illustration of a subset of thefactors, a model risk for the model, and a selectable feature associatedwith a selected factor in the subset and a selected time period;receiving, through the first graphical user interface, a selection ofthe selectable feature; and, in response to receiving the selection,generating a second graphical user interface, the second graphical userinterface recoloring at least a portion of the first graphical userinterface to highlight in the graphical illustration the selected factorduring the selected time period and comprising indications of thepredetermined criteria for the selected factor.

In another aspect, a system is disclosed that includes a processor anddata storage including instructions that, when executed by theprocessor, cause the system to perform operations. The operationsinclude receiving a model dataset generated for a model, the modeldataset comprising modeled values for a plurality of factors;identifying, based on the model dataset, primary modeled values andsecondary modeled values for the model and generating a first graphicaluser interface comprising a graphical illustration of a model risk forthe model, indications of the primary modeled values and the secondarymodeled values, and a first selectable feature associated with aselected primary modeled value; receiving through the first graphicaluser interface a selection of the first selectable feature; in responseto receiving the selection of the first selectable feature, generating asecond graphical user interface comprising a graphical illustration ofthe selected primary modeled value over a time period and a secondselectable feature associated with a model drift for the model;receiving through the second graphical user interface a selection of thesecond selectable feature; and, in response to receiving the selectionof the second selectable feature, generating a third graphical userinterface that overlays the second graphical user interface with agraphical illustration of the model drift, the graphical illustration ofthe model drift illustrating an extent to which the selected primarymodeled value contributes to the model drift relative to at least oneother primary modeled value.

Aspects of the disclosed embodiments may include non-transitory,tangible computer-readable media that store software instructions that,when executed by one or more processors, are configured for and capableof performing and executing one or more of the methods, operations, andthe like consistent with the disclosed embodiments. Also, aspects of thedisclosed embodiments may be performed by one or more processors thatare configured as special-purpose processor(s) based on softwareinstructions that are programmed with logic and instructions thatperform, when executed, one or more operations consistent with thedisclosed embodiments.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the disclosed embodiments, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate disclosed embodiments and,together with the description, serve to explain the disclosedembodiments. In the drawings:

FIG. 1 is a block diagram of an exemplary system, consistent withdisclosed embodiments.

FIG. 2 is a block diagram of an exemplary modeling system, consistentwith disclosed embodiments.

FIG. 3 is a block diagram of an exemplary monitoring system, consistentwith disclosed embodiments.

FIG. 4 is a flowchart of an exemplary dynamic monitoring process,consistent with disclosed embodiments.

FIGS. 5A-5C illustrate exemplary graphical user interfaces in a dynamicmonitoring process, consistent with disclosed embodiments.

FIG. 6 is a flowchart of another exemplary dynamic monitoring process,consistent with disclosed embodiments.

FIGS. 7A-7C illustrate exemplary graphical user interfaces in a dynamicmonitoring process, consistent with disclosed embodiments.

DETAILED DESCRIPTION

Reference will now be made in detail to the disclosed embodiments,examples of which are illustrated in the accompanying drawings.

The disclosed systems, methods, and media describe systems and methodsfor dynamic monitoring through graphical user interfaces. In someembodiments, the graphical user interfaces may be used for dynamicmonitoring of one or more models.

A dynamically monitored model may be, for example, a model generated bysoftware, an application, and/or other instructions. In someembodiments, the model may take the form of and/or include a modeldataset. The model data set may include, for example, modeled values forany number of factors.

The factors and the modeled values in the dataset may take any number offorms. For example, for a model modeling the mortgage industry, thefactors may include home prices, housing volatility, mortgage payments,interest rates, demographics, and/or foreclosures. As another example,for a model modeling consumer credit, the factors may include creditscores, debt-to-income ratios, payment histories, defaults, and/ordemographics. Other factors are possible as well. Modeled values may beany values generated by the model, based on the factors. For example, amodeled value for a factor may take the form of a single value and/or avalue over time. As another example, a modeled value for a factor maytake the form of a distribution or other statistical calculationassociated with the factor, such as a performance or a populationstability index for the factor. Other modeled values are possible aswell. In general, the modeled values may be independent of one anotheror may be interdependent. In some embodiments, the model may considermultiple factors in modeling a factor to generate a modeled value.

FIG. 1 is a block diagram of an exemplary system 100, consistent withdisclosed embodiments. System 100 may be configured for performing adynamic monitoring process consistent with disclosed embodiments.

As shown, system 100 may include a model system 102, a monitoring system104, data storage 106, and a display device 108, all of which may becommunicatively coupled by a network 114. As shown, model system 102 mayinclude one or more model dataset(s) 110, and data storage 106 mayinclude one or more input file(s) 112. While only one model system 102,monitoring system 104, data storage 106, and display device 108 areshown, it will be understood that system 100 may include more than onemodel system 102, monitoring system 104, data storage 106 and/or displaydevice 108 as well. Further, while certain numbers of model dataset(s)110 and input file(s) 112 are shown, it will be understood that system100 may include more or fewer of these components as well. Moregenerally, the components and arrangement of the components included insystem 100 may vary. Thus, system 100 may include other components thatperform or assist in the performance of one or more processes consistentwith the disclosed embodiments.

Model system 102 may be one or more computing devices configured togenerate model dataset(s) 110. Each of model dataset(s) 110 may includemodeled values for each of a number of factors. The factors and modeledvalues in a model dataset 110 may vary depending on the nature of themodel. For example, for a model modeling mortgages, the factors mayinclude factors associated with mortgages, such as interest rates,mortgage trade lines, public records, etc., and the modeled values mayinclude values determined according to the model for each of thefactors. As another example, for a model modeling automobile financing,the factors may include factors associated with automobile financing,such as pricing information, automobile makes and models, and/ordepreciation information, and the modeled values may include valuesdetermined according to the model for each of the factors. The modeledvalues may be included in the model dataset 110. In some embodiments,the model dataset 110 may include the factors as well.

Once a model dataset 110 has been generated, model system 102 mayprovide the model dataset 110 to monitoring system 104. In someembodiments, model system 102 may maintain model dataset 110 in a datastorage location accessible by monitoring system 104, such as in localdata storage at model system 102 and/or in remote data storage, such asdata storage 106. Alternatively or additionally, model system 102 mayprovide the model dataset 110 to monitoring system 104 over network 114or another communication channel. For instance, model system 102 mayprovide the model dataset 110 through a “push” mechanism, upon request,and/or periodically.

In connection with the dynamic monitoring processes, monitoring system104 may maintain one or more input file(s) 112. While the input file(s)112 are shown to be stored in data storage 106, in some embodimentsinput file(s) 112 may alternatively or additionally be stored in localdata storage at monitoring system 104 and/or in other remote datastorage accessible over network 114. Monitoring system 104 may be one ormore computing devices configured to carry out the dynamic monitoringprocesses described herein.

Input files 112 may include indications of predetermined criteria forthe factors used by model system 102 to generate model dataset(s) 110.The predetermined criteria for a factor may take the form of and/orinclude, for example, a target value for the factor and/or a range oftarget values for the factor, a historical value for the factor and/or arange of historical values for the factor, a distribution of values forthe factor, and/or one or more predetermined rules governing arelationship between the factor and the modeled value, such as adeviation between the factor and the modeled value and/or a factor riskassociated with the deviation between the factor and the modeled value.Input files 112 may take other forms as well.

In connection with the dynamic monitoring processes, monitoring system104 may additionally receive one or more model dataset(s) 110 from modelsystem 102. In some embodiments, model system 102 may maintain modeldataset 110 in a data storage location accessible by monitoring system104, such as in local data storage at model system 102 and/or in remotedata storage, such as data storage 106. Alternatively or additionally,model system 102 may provide the model dataset 110 to monitoring system104 over network 114 or another communication channel. For instance,model system 102 may provide the model dataset 110 through a “push”mechanism, upon request, and/or periodically.

Based on model dataset 110 and input file 112, monitoring system 104 maygenerate one or more graphical user interfaces. The graphical userinterfaces may be dynamic graphical user interfaces with which a usermay interact to dynamically isolate, identify, and illustrate factorsthat contribute to model drift and/or model risk. The graphical userinterfaces may include, for example, graphical and/or textualillustrations of one or more of the factors, the modeled values, and/orthe model. Alternatively or additionally, the graphical user interfacesmay include graphical and/or textual illustrations of informationassociated with the factors, modeled values, and/or model, such asfactor risk, model risk, and/or model drift. Example graphical userinterfaces are described below in connection with FIGS. 5A-5C and 7A-7C.

In some embodiments, graphical user interfaces generated by monitoringsystem 104 may be provided for display on, for example, a display device108. Display device 108 may be any device configured to displaygraphical user interfaces, including but not limited to monitors,desktop computers, laptop computers, tablets, and other computingdevices. Other display devices are possible as well.

Data storage 106 may include one or more computing devices configured tomaintain information for use in the dynamic monitoring processesdescribed herein. For example, as shown, data storage 106 may maintaininput file(s) 112. In some embodiments, input file(s) 112 may bemodifiable in data storage 106 by one or both of model system 102 andmonitoring system 104. In some embodiments, data storage 106 maymaintain other information as well, such as model dataset(s) 110 and/orother information.

In some embodiments, data storage 106 may take the form of one or moreservers or databases, such as Oracle™ databases, Sybase™ databases, orother relational databases or non-relational databases, such as Hadoopsequence files, HBase, or Cassandra. Such database(s) may includecomputing components (e.g., database management system, database server,etc.) configured to receive and process requests for data stored inmemory devices of the database(s) and to provide data from thedatabase(s). Alternatively or additionally, data storage 106 may includecloud-based storage accessible by model system 102 and/or monitoringsystem 104 over network 114 and/or another network.

In some embodiments, data storage 106 may be configured to aggregateinformation from one or more sources, such as one or more servers innetwork 114 and/or system 100. Alternatively or additionally, datastorage 106 may be included in and/or otherwise associated with one ormore such sources. In some embodiments, data storage 106 may aggregatedata from, may be included in, and/or may be otherwise associated with afinancial service entity that provides, maintains, manages, or otherwiseoffers financial services. For example, the financial service entity maybe a bank, credit card issuer, or any other type of financial serviceentity that generates, provides, manages, and/or maintains user accountsfor customers. In some embodiments, user accounts may include, forexample, credit card accounts, loan accounts, checking accounts, savingsaccounts, reward or loyalty program accounts, and/or any other type offinancial service account. While data storage 106 is shown separately,in some embodiments data storage 106 may be included in and/or otherwiseassociated with model system 102, monitoring system 104, and/or anotherentity in network 114 and/or system 100.

Network 114 may be any type of network configured to providecommunication between components of system 100. For example, network 114may be any type of network (including infrastructure) that providescommunications, exchanges information, and/or facilitates the exchangeof information, such as the Internet, a Local Area Network, near fieldcommunication (NFC), or other suitable connection(s) that enables thesending and receiving of information between the components of system100. In other embodiments, one or more components of system 100 maycommunicate directly through a dedicated communication link(s).

It is to be understood that the configuration and boundaries of thefunctional building blocks of system 100 have been defined herein forthe convenience of the description. Alternative boundaries may bedefined so long as the specified functions and relationships thereof areappropriately performed. Alternatives (including equivalents,extensions, variations, deviations, etc., of those described herein)will be apparent to persons skilled in the relevant art(s) based on theteachings contained herein. Such alternatives fall within the scope andspirit of the disclosed embodiments.

FIG. 2 is a block diagram of an exemplary model system 200, consistentwith disclosed embodiments. As shown, model system 200 may include acommunication device 202, one or more processor(s) 204, and memory 206including one or more program(s) 208 and data 210.

Model system 200 may take the form of a server, general purposecomputer, mainframe computer, or any combination of these components.Other implementations consistent with disclosed embodiments are possibleas well. Model system 200 may, for example, be similar to model system102 described above.

Communication device 202 may be configured to communicate with one ormore entities. For example, in some embodiments, communication device202 may be configured to communicate with a monitoring system and/ordata storage, such as monitoring system 104 and data storage 106described above. In some embodiments, communication device 202 may beconfigured to communicate with the monitoring system and/or data storagethrough a network, such as network 114 described above. Communicationdevice 202 may communicate with the monitoring system and/or datastorage in other manners as well.

Communication device 202 may be configured to communicate with themonitoring system to, for example, provide one or more model dataset(s)214 to the monitoring system. In some embodiments, model system 200 maymaintain the model dataset(s) in data 210, and communication device 202may permit the monitoring system to access the model dataset(s) 214 indata 210. Alternatively or additionally, communication device 202 may beconfigured to communicate with the monitoring system to provide themodel dataset(s) 214 to the monitoring system over a network, such asnetwork 114 described above, or another communication channel. Stillalternatively or additionally, communication device 202 may beconfigured to communicate with remote data storage, such as data storage106 described above, to provide the model dataset(s) 214 to the datastorage where the model dataset(s) 214 may be accessed by the monitoringdevice.

Communication device 202 may also be configured to communicate withother components. In general, communication device 202 may be configuredto provide communication over a network, such as network 114 describedabove. To this end, communication device 202 may include, for example,one or more digital and/or analog devices that allow model system 200 tocommunicate with and/or detect other components, such as a networkcontroller and/or wireless adaptor for communicating over the Internet.Other implementations consistent with disclosed embodiments are possibleas well.

Processor(s) 204 may include one or more known processing devices, suchas a microprocessor from the Core™, Pentium™ or Xeon™ familymanufactured by Intel™, the Turion™ family manufactured by AMO™, the“Ax” or “Sx” family manufactured by Apple™, or any of various processorsmanufactured by Sun Microsystems, for example. The disclosed embodimentsare not limited to any type of processor(s) otherwise configured to meetthe computing demands required of different components of model system200.

Memory 206 may include one or more storage devices configured to storeinstructions used by processor(s) 204 to perform functions related todisclosed embodiments. For example, memory 206 may be configured withsoftware instructions, such as program(s) 208, that may perform one ormore operations when executed by processor(s) 204. The disclosedembodiments are not limited to separate programs or computers configuredto perform dedicated tasks. For example, memory 206 may include a singleprogram 208 that performs the functions of model system 200, orprogram(s) 208 may comprise multiple programs. Memory 206 may also storedata 210 that is used by program(s) 208. In some embodiments, forexample, data 210 may include information for use in a modeling process,such as factors, modeled values, models, and/or any informationassociated with factors, modeled values, and/or models, such as factorrisk, model risk, and/or model drift. Other data 210 is possible aswell.

In certain embodiments, memory 206 may store sets of instructions forcarrying out a modeling process for generating model dataset(s) 214. Forexample, as shown, memory 206 may include a modeling program 212configured to generate the model dataset(s) 214 for use in a dynamicmonitoring process, such as the dynamic monitoring processes describedbelow in connection with FIGS. 4 and 6 . Any number of modeling programs212 are possible, and the disclosed dynamic monitoring processes may beperformed using any modeling program that generates a model dataset 214including modeled values for factors. Other instructions are possible aswell. In general, instructions may be executed by processor(s) 204 toperform one or more processes consistent with disclosed embodiments.

The components of model system 200 may be implemented in hardware,software, or a combination of both hardware and software, as will beapparent to those skilled in the art. For example, although one or morecomponents of model system 200 may be implemented as computer processinginstructions, all or a portion of the functionality of model system 200may be implemented instead in dedicated electronics hardware.

FIG. 3 is a block diagram of an exemplary monitoring system 300,consistent with disclosed embodiments. As shown, monitoring system 300may include a communication device 302, one or more processor(s) 304,and memory 306 including one or more program(s) 308 and data 310.

Monitoring system 300 may take the form of a server, general purposecomputer, mainframe computer, or any combination of these components.Other implementations consistent with disclosed embodiments are possibleas well. Monitoring system 300 may, for example, be similar tomonitoring system 104 described above.

Communication device 302 may be configured to communicate with one ormore entities. For example, in some embodiments, communication device302 may be configured to communicate with a model system and/or datastorage, such as model system 102 and data storage 106 described above.In some embodiments, communication device 302 may be configured tocommunicate with the model system and/or data storage through a network,such as network 114 described above. Communication device 302 maycommunicate with the model system and/or data storage in other mannersas well.

Communication device 302 may be configured to communicate with the modelsystem to, for example, receive one or more model dataset(s), such asmodel dataset(s) 112 and 214 described above. In some embodiments, themodel system may maintain the model dataset(s) in local data storage atthe model system, and communication device 302 may permit monitoringsystem 300 to access the model dataset(s) at the model system.Alternatively or additionally, communication device 302 may beconfigured to communicate with the model system to receive the modeldataset(s) to the monitoring system over a network, such as network 114described above, or another communication channel. Still alternativelyor additionally, the model system may provide the model dataset(s) toremote data storage, such as data storage 106 described above, andcommunication device 302 may be configured to communicate with datastorage 106 to access the model dataset(s).

Communication device 302 may also be configured to communicate withother components. In general, communication device 302 may be configuredto provide communication over a network, such as network 114 describedabove. To this end, communication device 302 may include, for example,one or more digital and/or analog devices that allow monitoring system300 to communicate with and/or detect other components, such as anetwork controller and/or wireless adaptor for communicating over theInternet. Other implementations consistent with disclosed embodimentsare possible as well.

Processor(s) 304 may include one or more known processing devices, suchas a microprocessor from the Core™, Pentium™ or Xeon™ familymanufactured by Intel™, the Turion™ family manufactured by AMO™, the“Ax” or “Sx” family manufactured by Apple™, or any of various processorsmanufactured by Sun Microsystems, for example. The disclosed embodimentsare not limited to any type of processor(s) otherwise configured to meetthe computing demands required of different components of monitoringsystem 300.

Memory 306 may include one or more storage devices configured to storeinstructions used by processor(s) 304 to perform functions related todisclosed embodiments. For example, memory 306 may be configured withsoftware instructions, such as program(s) 308, that may perform one ormore operations when executed by processor(s) 304. The disclosedembodiments are not limited to separate programs or computers configuredto perform dedicated tasks. For example, memory 306 may include a singleprogram 308 that performs the functions of monitoring system 300, orprogram(s) 308 may comprise multiple programs. Memory 306 may also storedata 310 that is used by program(s) 308. In some embodiments, forexample, data 310 may include information for use in a dynamicmonitoring process, such as factors, modeled values, models, and/or anyinformation associated with factors, modeled values, and/or models, suchas factor risk, model risk, and/or model drift. For example, data 310may include any of the information described below in connection withdynamic monitoring processes 400 and 600. Other data 310 is possible aswell.

In certain embodiments, memory 306 may store sets of instructions, suchas dynamic monitoring program 312, for carrying out a dynamic monitoringprocess, such as the dynamic monitoring processes described below inconnection with FIGS. 4 and 6 . Other instructions are possible as well.In general, instructions may be executed by processor(s) 306 to performone or more processes consistent with disclosed embodiments.

The components of monitoring system 306 may be implemented in hardware,software, or a combination of both hardware and software, as will beapparent to those skilled in the art. For example, although one or morecomponents of monitoring system 300 may be implemented as computerprocessing instructions, all or a portion of the functionality ofmonitoring system 300 may be implemented instead in dedicatedelectronics hardware.

Monitoring system 300 may include more, fewer, and/or differentcomponents than those shown. For example, in some embodiments,monitoring system 300 may include and/or may be communicatively coupledto one or more display devices, such as display device 108 describedabove, configured to provide output and/or display graphical userinterfaces, such as the graphical user interfaces described below inconnection with FIGS. 5A-5C and 7A-7C. In some embodiments, such adisplay device may include a screen for displaying a graphical and/ortext-based user interface, including but not limited to, liquid crystaldisplays (LCD), light emitting diode (LED) screens, organic lightemitting diode (OLEO) screens, and other known display devices. Asanother example, in some embodiments monitoring system 200 may includeand/or may be communicatively coupled to one or more digital and/oranalog devices configured to receive input, such as a touch-sensitivearea, keyboard, buttons, or microphones. Other components are possibleas well.

FIG. 4 is a flowchart of an exemplary dynamic monitoring process 400,consistent with disclosed embodiments. Dynamic monitoring process 400may be carried out by a monitoring system, such as monitoring systems104 and 300 described above. FIG. 4 will be explained with reference toFIGS. 5A-5C, which illustrate exemplary graphical user interfaces in adynamic monitoring process, consistent with disclosed embodiments.

As shown in FIG. 4 , dynamic monitoring process 400 begins at step 402with maintaining an input file including predetermined criteria for aplurality of factors. The factors and the predetermined criteria maytake any of the forms described above.

In some embodiments, the factors may include any variable relevant to amodel. For example, a factor may be any quantifiable value usable by amodel system to generate a model. In some embodiments, the factors mayvary depending on the nature of the model. For example, for a modelmodeling some aspect of the housing industry, the factors may includequantifiable values relevant to the housing industry, such as housingprices, interest rates, demographic information, housing volatility,foreclosures, payments, etc. As another example, for a model modelingsome aspect of the automobile industry, the factors may includequantifiable values relevant to the automobile industry, such asautomobile sales, automobile loan statistics, credit scores,demographics, payments, etc. Other factors and other models are possibleas well.

In some embodiments, the predetermined criteria may include any criteriafor evaluating a modeled value. For example, predetermined criteria maybe any criteria for determining whether a modeled value contributes tomodel drift and/or model risk. For example, the predetermined criteriafor a factor may take the form of and/or include a target value for thefactor and/or a range of target values for the factor. The predeterminedcriteria may be used to evaluate a modeled value by, for instance,determining whether the modeled value is consistent with the targetvalue, deviates by an acceptable amount from the target value, is withinthe range of target values, etc. As another example, the predeterminedcriteria for a factor may take the form of and/or include a historicalvalue for the factor and/or a range of historical values for the factor.The predetermined criteria may be used to evaluate a modeled value by,for instance, determining whether the modeled value is consistent withthe historical value, deviates an acceptable amount from the historicalvalue, is within the range of historical values, etc. As still anotherexample, the predetermined criteria for a factor may take the form of adistribution of values for the factor, and the predetermined criteriamay be used to evaluate a modeled value by, for instance, determiningwhether the modeled value falls within the distribution. As yet anotherexample, the predetermined criteria for a factor may take the form ofone or more rules governing a relationship between the factor and themodeled value. For example, the predetermined criteria may be a rulepermitting only a certain degree of deviation from a historical valueover a period of time. As another example, the predetermined criteriamay be a rule designed to identify an increasing deviation between themodeled value and a target value for the factor. Other predeterminedcriteria are possible as well.

In some embodiments, maintaining the input file may involve, forexample, maintaining the input file in data storage, such as datastorage 106 described above. In some embodiments, the data storage maybe accessible by, for example, a remote entity in addition to themonitoring system. The remote entity may be configured to, for example,modify the input file. For instance, the remote entity may modify theinput file to modify the factors and/or predetermined criteria indicatedin the input file.

Dynamic monitoring process 400 continues at step 404 with receiving amodel dataset generated at a model system, where the model datasetincludes modeled values for the factors. The model dataset may bereceived from a model system, such as model systems 102 and 200described above. In some embodiments, the monitoring system may receivethe model data set via a communication device, such as communicationdevice 302 described above.

The model dataset may take any of the forms described above for modeldataset(s) 110 and 214. In general, the model dataset may be anycollection of modeled values for the factors. The modeled values may beany values generated by the model for the factors. For example, amodeled value for a factor may take the form of a single value and/or avalue over time. As another example, a modeled value for a factor maytake the form of a distribution or other statistical calculationassociated with the factor, such as a performance or a populationstability index for the factor. Other modeled values are possible aswell. In general, the modeled values may be independent of one anotheror may be interdependent. In some embodiments, the model may considermultiple factors in modeling a factor to generate a modeled value.

Dynamic monitoring process 400 continues at step 406 with generating,based on the input file and the model dataset, a first graphical userinterface comprising a graphical illustration of a subset of thefactors, a model risk for the model, and a selectable feature associatedwith a selected factor in the subset and a selected time period.

The subset of the factors may include fewer than all of the factors. Insome embodiments, the monitoring system may identify the subset from thefactors based on factor risks posed by the factors. For example, themonitoring system may use the predetermined criteria to ascertain afactor risk posed by each of the factors, and the monitoring system mayidentify for the subset factors exhibiting a relatively high factorrisk. Alternatively or additionally, the monitoring system may identifythe subset from the factors based on an extent to which each factorcontributes to model drift and/or model risk. For example, themonitoring system may use the predetermined criteria to ascertain eachfactor's impact on the model drift and/or model risk, and the monitoringsystem may identify for the subset factors exhibiting a relatively highcontribution to the model drift and/or model risk.

In some embodiments, to ascertain the factor risk and/or thecontributions to model drift and/or model risks of a factor, themonitoring system may apply the predetermined criteria in the input fileto the modeled values. For example, in some embodiments, the monitoringsystem may, for each factor, compare the modeled value for the factor tothe predetermined criteria for the factor to determine a deviation ofthe modeled value. The deviation may be, for example, a deviation of themodeled value from a target value, a range of target values, ahistorical value, a range of historical values, a distribution, etc.,specified by the predetermined criteria, and the monitoring system mayuse the deviation to assess the factor risk and/or the contribution tomodel drift and/or model risk of a factor. For instance, if thedeviation exceeds the predetermined threshold, indicating that themodeled value deviates significantly from the predetermined criteria,the monitoring system may determine that the factor poses a relativelysignificant factor risk and/or is contributing relatively significantlyto model drift and/or model risk. Accordingly, the monitoring system mayidentify the factor for the subset. On the other hand, if the deviationdoes not exceed the predetermined threshold, indicating that the modeledvalue does not deviate significantly from the predetermined criteria,the monitoring system may determine that the factor poses a relativelyinsignificant factor risk and/or is contributing relativelyinsignificantly to model drift and/or model risk. Accordingly, themonitoring system may decline to identify the factor for the subset.

Alternatively or additionally, in some embodiments the monitoring systemmay identify factors for the subset based on factor risk. For instance,if the determined factor risk for a factor exceeds a predeterminedthreshold, indicating that the factor poses a relatively significantfactor risk and/or is contributing relatively significantly to modeldrift and/or model risk, the monitoring system may identify the factorfor the subset. On the other hand, if the determined factor risk doesnot exceed the predetermined threshold, indicating that the factor posesa relatively insignificant factor risk and/or is contributing relativelyinsignificantly to model drift and/or model risk, the monitoring systemmay decline to identify the factor for the subset.

An example first graphical user interface 500 is shown in FIG. 5A. Asshown, first graphical user interface 500 may include a graphicalillustration 502 of the subset of the factors. Graphical illustration502 may take any form that graphically illustrates one or more aspectsof the subset of the factors. For example, in some embodiments,graphical illustration 502 may take the form of a chart or graphillustrating quantitative values associated with the subset of thefactors. For example, graphical illustration 502 may illustrate a factorrisk for each factor in the subset. As another example, graphicalillustration 502 may illustrate a population stability index, adistribution, performance, or other quantitative value associated witheach factor in the subset of the factors. Alternatively or additionally,graphical illustration 502 may illustrate quantitative values associatedwith the subset of factors over one or more time periods. For example,graphical illustration 502 may illustrate a factor risk for each factorover one or more time periods. Still alternatively or additionally, insome embodiments, graphical illustration 502 may take the form of amixed textual and graphical illustration including textual information,such as textual information identifying the factors, textual informationidentifying first graphical user interface 500 may include a list orother textual description including indications of each of the subset ofthe factors, indications of the one or more time periods, and/or othertextual information. While graphical illustration 502 is shown in asingle color, in some embodiments color may be used to enhance graphicalillustration 502. For example, the factors illustrated in graphicalillustration 502 may be color-coded to distinguish among the factors,quantitative values, time periods, textual information, etc.

In some embodiments, first graphical user interface 500 may furtherillustrate a model risk for the model. The model risk may be determinedbased on, for example, factor risks for factors in the subset, which mayin turn be determined based on the modeled values and the predeterminedcriteria, as described above. In some embodiments, the model risk may beillustrated through a chart or graph illustrating quantitative valuesassociated with the model risk and/or the subset of factors. Forexample, the model risk may be illustrated through a performance graphillustrating a performance of the model relative to other models thatgenerate modeled values for the subset of factors. As another example,the model risk may be illustrated through a population stability index,a distribution, or other quantitative value associated with the modelrisk and/or factor risks for the subset of factors. Alternatively oradditionally, the graphical illustration of the model risk mayillustrate an extent to which each factor in the subset contributes tothe model risk. Still alternatively or additionally, the graphicalillustration of the model risk may illustrate a deviation of the modelfrom reality over time. The model risk may be illustrated in othermanners as well.

An example graphical illustration 504 of the model risk is shown in FIG.5A. As shown, graphical illustration 504 takes the form of a graph thatillustrates the model risk over a number of time periods, labeled “01,”“02,” etc. In some embodiments, each of the lines shown in graphicalillustration 504 may correspond to a factor in the subset. Other examplegraphical illustrations of the model risk are possible as well.

In some embodiments, instead of or in addition to graphical illustration504 of the model risk, first graphical user interface 500 may include atextual description 510 of the model risk. The textual description mayinclude, for example, a description of the subset of factors, the factorrisk for each factor in the subset of factors, the time periods, and/oran extent to which each factor in the subset contributes to the modelrisk. Other textual description is possible as well.

First graphical user interface 500 may further include a selectablefeature 506, as shown. Selectable feature 506 may, for example, beassociated with a particular factor from the input file. For example, asshown, first graphical user interface 500 may include an indication ofeach factor in the subset, and the selectable feature may take the formof an indication of a factor. For example, as shown, selectable feature506 is associated with “Factor 3.” In some embodiments, where firstgraphical user interface 500 includes indications of the factors in thesubset, the factors may be ranked and/or otherwise presented accordingto the extent to which they contribute to the model risk, a severity ofthe factor risk for each factor, and/or other criteria. Alternatively oradditionally, in some embodiments, selectable feature 506 may beassociated with a time period, such as a time, day, or period of days.Still alternatively or additionally, in some embodiments selectablefeature 506 may, upon selection, present additional options for inputfrom a user. For example, selection of selectable feature 506 maypresent a user with options to, for instance, “Keep Only” the selectedfactor or to “Exclude” the selected factor. Such additional options may,for example, be presented as overlay to first graphical user interface500, such as in a pop-up window. In general, selectable feature 506 maybe any feature configured to be selected by a user through firstgraphical user interface 500.

In some embodiments, first graphical user interface 500 may include oneor more additional features, such as additional feature 508. Additionalfeatures of first graphical user interface 500 may include textualdescriptions and/or graphical illustrations of one or more additionalaspects of the model, the model risk, the model drift, the subset offactors, the factor risk, and/or the modeled values. Any additionalfeature 508 may take any of the forms described above for graphicalillustrations 502, 504 and textual description 510.

Returning to FIG. 4 , at step 406 dynamic monitoring process 400includes receiving through the first graphical user interface aselection of the selectable feature. For example, as shown in FIG. 5A, auser may select selectable feature 506 by providing input to graphicaluser interface 500. The input may take the form of, for example, a mouseclick, a key stroke, or a touchscreen touch. Other inputs are possibleas well.

Dynamic monitoring process 400 continues at step 408 with, in responseto receiving the selection, generating a second graphical userinterface, the second graphical user interface recoloring at least aportion of the first graphical user interface to highlight in thegraphical illustration the selected factor.

An example second graphical user interface 512 is shown in FIG. 5B. Asshown, selectable feature 506 for “Factor 3” has been selected, andgraphical illustration 502 of the subset of factors has been recoloredto highlight, in graphical illustration 502, the selected factor,namely, “Factor 3.” While the recoloring is shown as a balding of theline associated with the selected factor, it will be understood that theselected factor may be highlighted in graphical illustration 502 inother manners as well, including through the use of colors, patterns,overlays, etc.

Alternatively or additionally, in some embodiments second graphical userinterface 512 may add and/or replace one or more features of firstgraphical user interface 500 with additional information pertaining tothe selected factor. For example, as shown, second graphical userinterface 512 replaces textual description 510 of the model risk with atextual description 516 pertaining to the selected factor, “Factor 3.”Textual description 516 pertaining to the selected factor may include,for example, a description of the selected factor, the factor risk forthe selected factor, one or more time periods associated with theselected factor, and/or an extent to which the selected factorcontributes to the model risk. Other textual description is possible aswell. As another example, as shown, second graphical user interface 512may add a second selectable feature 514. The second selectable feature514 may permit a user to view additional details associated with theselected factor. For example, as shown, the second selectable feature514, if selected, may provide a more detailed illustration of theselected factor during the selected time period. Other selectablefeatures are possible as well.

In general, selectable features may facilitate navigation through aseries of graphical user interfaces through which a user may isolate,identify, and view illustrations of the selected factor, the factor riskfor the selected factor, the model risk, the model drift, and/or anextent to which the selected factor contributes to the model risk and/orthe model drift. In some embodiments, for example, second graphical userinterface 512 may illustrate the deviation for the selected factordetermined by comparing the modeled value for the factor to thepredetermined criteria, as described above. As another example, thesecond graphical user interface 512 may illustrate the factor risk forthe selected factor determined by comparing the modeled value for thefactor to the predetermined criteria, as described above.

As still another example, a third graphical user interface 518, as shownin FIG. 5C, may add as an overlay to second graphical user interface 512a description associated with the recoloring. For example, the thirdgraphical user interface 518 may include a pop-up window 520 describing,for example, the predetermined criteria for the selected factor, themodeled value for the selected factor, the factor risk for the selectedfactor, and/or an extent to which the selected factor contributes tomodel drift and/or model risk.

FIG. 6 is a flowchart of another exemplary dynamic monitoring process600, consistent with disclosed embodiments. Dynamic monitoring process600 may be carried out by a monitoring system, such as monitoringsystems 104 and 300 described above. FIG. 6 will be explained withreference to FIGS. 7A-5C, which illustrate exemplary graphical userinterfaces in a dynamic monitoring process, consistent with disclosedembodiments.

As shown, dynamic monitoring process 600 begins at step 602 withreceiving a model dataset generated for a model, where the model datasetincludes modeled values for a plurality of factors. The model dataset,the modeled values, and the plurality of factors may take any of theforms described above, and the model dataset may be received in any ofthe manners described above.

Dynamic monitoring process 600 continues at step 604 with, based on themodel dataset, identifying primary modeled values and secondary modeledvalues for the model and generating a first graphical user interface.The graphical user interface may include a graphical illustration of amodel risk for the model, indications of the primary modeled values andthe secondary modeled values, and a first selectable feature associatedwith a selected primary modeled value.

An example first graphical user interface 700 is shown in FIG. 7A. Asshown, first graphical user interface 700 may include a graphicalillustration 702 of the model risk for the model. The model risk may bedetermined based on, for example, the factor risks for factors in thesubset, which may in turn be determined based on the modeled values andthe predetermined criteria, as described above. In some embodiments, themodel risk may be illustrated through a chart or graph illustratingquantitative values associated with the model risk and/or the subset offactors, such as graphical illustration 702. For example, the model riskmay be illustrated through a performance graph illustrating aperformance of the model relative to other models that generate modeledvalues for the subset of factors. As another example, the model risk maybe illustrated through a population stability index, a distribution, orother quantitative value associated with the model risk and/or factorrisks for the subset of factors. Alternatively or additionally, thegraphical illustration of the model risk may illustrate an extent towhich each factor in the subset contributes to the model risk. Stillalternatively or additionally, the graphical illustration of the modelrisk may illustrate a deviation of the model from reality over time. Themodel risk may be illustrated in other manners as well.

Further, as shown, first graphical user interface 700 may include anindication 704 of the primary modeled values and an indication 706 ofthe secondary modeled values. In some embodiments, the monitoring systemmay identify the primary modeled values in a manner similar to that inwhich the monitoring system identified the factors in the subset, asdescribed above. For example, in some embodiments, the monitoring systemmay maintain an input file including predetermined criteria for thefactors, as described above. For example, the input file may bemaintained in data storage, such as data storage 106 described above.

The monitoring system may compare the modeled values for the factors inthe model dataset to the predetermined criteria for the factors todetermine a deviation, as described above. If the deviation exceeds apredetermined threshold, indicating that a modeled value deviatessignificantly from the predetermined criteria, the monitoring system maydetermine that the modeled value poses a relatively significant factorrisk and/or is contributing relatively significantly to model driftand/or model risk. Accordingly, the monitoring system may identify themodeled value as a primary modeled value. On the other hand, if thedeviation does not exceed the predetermined threshold, indicating thatthe modeled value does not deviate significantly from the predeterminedcriteria, the monitoring system may determine that the modeled valueposes a relatively insignificant factor risk and/or is contributingrelatively insignificantly to model drift and/or model risk.Accordingly, the monitoring system may identify the modeled value as asecondary modeled value.

Alternatively or additionally, in some embodiments the monitoring systemmay identify primary and second modeled values based on factor risk. Forinstance, if the determined factor risk for a modeled value exceeds apredetermined threshold, indicating that the modeled value poses arelatively significant factor risk and/or is contributing relativelysignificantly to model drift and/or model risk, the monitoring systemmay identify the modeled value as a primary modeled value. On the otherhand, if the determined factor risk does not exceed the predeterminedthreshold, indicating that the modeled value poses a relativelyinsignificant factor risk and/or is contributing relativelyinsignificantly to model drift and/or model risk, the monitoring systemmay identify the modeled value as a secondary modeled value.

In some embodiments, where first graphical user interface 700 includesindications of the primary and secondary modeled values, the modeledvalues may be ranked and/or otherwise presented according to the extentto which they contribute to the model risk, a severity of the factorrisk for each factor, and/or other criteria.

Returning to FIG. 7A, first graphical user interface 700 may furtherinclude a first selectable feature 708 associated with a selectedprimary modeled value, as shown. The first selectable feature 708 maytake any of the forms described above. In some embodiments, the firstselectable feature 708 may take the form of a request to isolate theprimary modeled value. Alternatively or additionally, in someembodiments, the first selectable feature 708 may take the form of arequest for a detailed view of the selected primary model. Otherselectable features are possible as well.

In some embodiments, first graphical user interface 700 may furtherinclude one or more additional features, such as graphical illustration710 and textual description 712, each of which may take any of the formsdescribed above for features in connection with FIG. 4 .

Returning to FIG. 6 , at step 606 dynamic monitoring process 600 furtherincludes receiving through the first graphical user interface aselection of the first selectable feature. The first selectable featuremay be selected in any of the manners described above.

Monitoring process 600 continues at step 608 with, in response toreceiving the selection of the first selectable feature, generating asecond graphical user interface. The second graphical user interface mayinclude a graphical illustration of the selected primary modeled valueover a time period and a second selectable feature associated with amodel drift for the model. In some embodiments, an indication of thetime period may be received through the first graphical user interface714 along with the first selectable feature 708.

An example second graphical user interface 714 is shown in FIG. 7B. Asshown, second graphical user interface 714 includes a graphicalillustration 716 of the selected primary modeled value. In someembodiments, graphical illustration 716 may present, for example, acomparison of the selected primary modeled value modeled by the modelwith modeled values for the factor generated using other availablemodels. Alternatively or additionally, in some embodiments graphicalillustration 716 may present, for example, a comparison of the selectedprimary modeled value among different segments of customers. While onlytwo lines are shown in graphical illustration 716, it will be understoodthat additional lines (for example, illustrating additional availablemodels and/or additional segments of customers) are possible as well.Graphical illustration 702 may take any of the forms described above inconnection with graphical illustration 502. In general, graphicalillustration 702 may take any form that graphically illustrates one ormore aspects of the subset of the factors. In some embodiments, inaddition to graphical illustration 702, second graphical user interface714 may include a description associated with the selected primarymodeled value over the time period.

Additionally, as shown, second graphical user interface 714 may includea second selectable feature 718. The second selectable feature 718 isassociated with a model drift for the model. The second selectablefeature 718 may take any the forms described above.

In some embodiments, second graphical user interface 714 may include oneor more additional features. For example, in some embodiments, thesecond graphical user interface 714 may include a graphical illustrationof the deviation for the selected primary modeled value. As anotherexample, the second graphical user interface 714 may illustrate the riskfor the selected primary model.

Returning to FIG. 6 , at step 610 dynamic monitoring process 600includes receiving through the second graphical user interface aselection of the second selectable feature. The second selectablefeature may be selected in any of the manners described above.

At step 612, in response to receiving the selection of the secondselectable feature, the monitoring system may generate a third graphicaluser interface. The third graphical user interface may include agraphical illustration of the model drift. For example, the graphicalillustration of the model drift may illustrate an extent to which theselected primary modeled value contributes to the model drift relativeto at least one other primary modeled value.

In some embodiments, monitoring system may maintain an input file, asdescribed above, and may determine the model drift associated with theselected primary modeled value based on the input file. For example,monitoring system may identify, based on the input file, a factorcorresponding to the selected primary modeled value. Further, monitoringsystem may identify, based on the input file, predetermined criteria forthe identified factor. Monitoring system may compare the selectedprimary modeled value to the predetermined criteria for the factor todetermine a deviation for the selected primary modeled value, asdescribed above. Based on the determined deviation, monitoring systemmay determine the model drift.

Alternatively or additionally, monitoring system may determine the modeldrift for the model generally, rather than only as associated with theselected primary modeled value. For example, monitoring system maycompare each factor to the predetermined criteria for the factor todetermine a deviation. Monitoring system may identify, based on theinput file, a factor corresponding to the selected primary modeledvalue. Monitoring system may further identify, based on the input file,predetermined criteria for the identified factor and determine the modeldrift for the model based on the determined deviations.

Still alternatively or additionally, monitoring system may determine themodel risk based on the primary modeled values and the predeterminedcriteria for the primary modeled values. Monitoring system may determinethe model risk in any of the manners described above.

FIG. 7C illustrates an example third graphical user interface 720. Asshown, third graphical user interface 720 overlays second graphical userinterface 714 with a graphical illustration 722 of the model drift. Thesecond graphical illustration 722 of the model drift may, for example,illustrate an extent to which the selected primary modeled valuecontributes to the model drift relative to one or more other primarymodeled values. Alternatively or additionally, second graphicalillustration 722 may take any of the forms described above in connectionwith the graphical illustrations of the model risk. It will beunderstood that the graphical user interfaces described above, includingtheir contents, are merely illustrative and are not meant to belimiting. That is, other graphical user interfaces, including theircontents, are possible as well.

In some examples, some or all of the logic for the above-describedtechniques may be implemented as a computer program or application or asa plug-in module or subcomponent of another application. The describedtechniques may be varied and are not limited to the examples ordescriptions provided.

Moreover, while illustrative embodiments have been described herein, thescope thereof includes any and all embodiments having equivalentelements, modifications, omissions, combinations (e.g., of aspectsacross various embodiments), adaptations and/or alterations as would beappreciated by those in the art based on the present disclosure. Forexample, the number and orientation of components shown in the exemplarysystems may be modified. Further, with respect to the exemplary methodsillustrated in the attached drawings, the order and sequence of stepsmay be modified, and steps may be added or deleted.

Thus, the foregoing description has been presented for purposes ofillustration only. It is not exhaustive and is not limiting to theprecise forms or embodiments disclosed. Modifications and adaptationswill be apparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. For example,while a financial service provider and merchant have been referred toherein for ease of discussion, it is to be understood that consistentwith disclosed embodiments other entities may provide such services inconjunction with or separate from a financial service provider andmerchant.

The claims are to be interpreted broadly based on the language employedin the claims and not limited to examples described in the presentspecification, which examples are to be construed as non-exclusive.Further, the steps of the disclosed methods may be modified in anymanner, including by reordering steps and/or inserting or deletingsteps.

Furthermore, although aspects of the disclosed embodiments are describedas being associated with data stored in memory and other tangiblecomputer-readable storage mediums, one skilled in the art willappreciate that these aspects may also be stored on and executed frommany types of tangible computer-readable media, such as secondarystorage devices, like hard disks, floppy disks, or CD-ROM, or otherforms of RAM or ROM. Accordingly, the disclosed embodiments are notlimited to the above described examples, but instead is defined by theappended claims in light of their full scope of equivalents.

1-20. (canceled)
 21. A system comprising: one or more processors andnon-transitory media storing instructions that, when executed by the oneor more processors, cause operation comprising: receiving a plurality ofmodeled values for a plurality of factors; generating a first graphicaluser interface comprising a first selectable feature, a first graphicalrepresentation related to model drift, and indications of a subset ofmodeled values of the plurality of modeled values; receiving, via thefirst graphical user interface, a first selection of the firstselectable feature, the first selectable feature associated with a firstmodeled value of the subset of modeled values; generating, based on thefirst selection, a second graphical user interface comprising a secondgraphical representation of the first modeled value associated with thefirst selectable feature over a time period; determining a deviation ofthe first modeled value based on one or more predetermined criteria fora first factor of the first modeled value; identifying, based on whetherthe deviation exceeds a predetermined threshold, the first modeled valueas having a first level of impact on the model drift; and presenting, onthe second graphical user interface, a third graphical representation ofan extent to which the first modeled value associated with the firstselectable feature contributes to the model drift.
 22. The system ofclaim 21, wherein identifying the first modeled value as having thefirst level of impact on the model drift comprises determining that thedeviation exceeds the predetermined threshold, and wherein presentingthe third graphical representation comprises presenting, based on thedetermination that the deviation exceeds the predetermined threshold,the third graphical representation of the extent to which the firstmodeled value associated with the first selectable feature contributesto the model drift.
 23. The system of claim 21, the operations furthercomprising: determining that the deviation exceeds the predeterminedthreshold, wherein identifying the first modeled value as having thefirst level of impact on the model drift comprises identifying the firstmodeled value as being a primary modeled value in lieu of being asecondary modeled value based on the determination that the deviationexceeds the predetermined threshold.
 24. The system of claim 21, theoperations further comprising: determining that the deviation does notexceed the predetermined threshold, wherein identifying the firstmodeled value as having the first level of impact on the model driftcomprises identifying the first modeled value as being a secondarymodeled value in lieu of being a primary modeled value based on thedetermination that the deviation does not exceed the predeterminedthreshold.
 25. A method comprising: receiving a plurality of modeledvalues for a plurality of factors; generating a first graphical userinterface comprising a first selectable feature, a first graphicalrepresentation related to model drift, and indications of a subset ofmodeled values of the plurality of modeled values; receiving, via thefirst graphical user interface, a first selection of the firstselectable feature, the first selectable feature associated with a firstmodeled value of the subset of modeled values; generating, based on thefirst selection, a second graphical user interface comprising a secondgraphical representation of the first modeled value associated with thefirst selectable feature over a time period; determining a deviation ofthe first modeled value based on one or more predetermined criteria fora first factor of the first modeled value; identifying, based on whetherthe deviation exceeds a predetermined threshold, the first modeled valueas having a first level of impact on the model drift; and causing, inconnection with the second graphical user interface, presentation of athird graphical representation of an extent to which the first modeledvalue associated with the first selectable feature contributes to themodel drift.
 26. The method of claim 25, wherein identifying the firstmodeled value as having the first level of impact on the model driftcomprises determining that the deviation exceeds the predeterminedthreshold, and wherein causing the presentation of the third graphicalrepresentation comprises causing, based on the determination that thedeviation exceeds the predetermined threshold, the presentation of thethird graphical representation of the extent to which the first modeledvalue associated with the first selectable feature contributes to themodel drift.
 27. The method of claim 25, further comprising: determiningthat the deviation exceeds the predetermined threshold, whereinidentifying the first modeled value as having the first level of impacton the model drift comprises identifying the first modeled value asbeing a primary modeled value in lieu of being a secondary modeled valuebased on the determination that the deviation exceeds the predeterminedthreshold.
 28. The method of claim 25, further comprising: determiningthat the deviation does not exceed the predetermined threshold, whereinidentifying the first modeled value as having the first level of impacton the model drift comprises identifying the first modeled value asbeing a secondary modeled value in lieu of being a primary modeled valuebased on the determination that the deviation does not exceed thepredetermined threshold.
 29. The method of claim 25, further comprisingdetermining the model drift based on deviations of each modeled value ofthe plurality of modeled values for each factor of the plurality offactors from a set of predetermined criteria for the factor.
 30. Themethod of claim 25, wherein causing the presentation of the thirdgraphical representation comprises overlaying, on the second userinterface, the third graphical representation of the extent to which thefirst modeled value associated with the first selectable featurecontributes to the model drift.
 31. The method of claim 25, whereinpresenting the third graphical representation comprises presenting apop-up window describing predetermined criteria for a correspondingfactor for the first modeled value.
 32. The method of claim 25, whereinthe first modeled value is a primary modeled value, and wherein theextent to which the first modeled value contributes to the model driftis determined relative to one or more other primary modeled values. 33.One or more non-transitory computer-readable media storing instructionsthat, when executed by one or more processors, cause operationscomprising: receiving a plurality of modeled values for a plurality offactors; generating a first graphical user interface comprising a firstselectable feature, a first graphical representation related to modeldrift, and indications of a subset of modeled values of the plurality ofmodeled values; receiving, via the first graphical user interface, afirst selection of the first selectable feature, the first selectablefeature associated with a first modeled value of the subset of modeledvalues; generating, based on the first selection, a second graphicaluser interface comprising a second graphical representation of the firstmodeled value associated with the first selectable feature over a timeperiod; determining a deviation of the first modeled value based on oneor more predetermined criteria for a first factor of the first modeledvalue; identifying, based on whether the deviation exceeds apredetermined threshold, the first modeled value as having a first levelof impact on the model drift; and causing, in connection with the secondgraphical user interface, presentation of a third graphicalrepresentation of an extent to which the first modeled value associatedwith the first selectable feature contributes to the model drift. 34.The one or more non-transitory computer-readable media of claim 33,wherein identifying the first modeled value as having the first level ofimpact on the model drift comprises determining that the deviationexceeds the predetermined threshold, and wherein causing thepresentation of the third graphical representation comprises causing,based on the determination that the deviation exceeds the predeterminedthreshold, the presentation of the third graphical representation of theextent to which the first modeled value associated with the firstselectable feature contributes to the model drift.
 35. The one or morenon-transitory computer-readable media of claim 33, the operationsfurther comprising: determining that the deviation exceeds thepredetermined threshold, wherein identifying the first modeled value ashaving the first level of impact on the model drift comprisesidentifying the first modeled value as being a primary modeled value inlieu of being a secondary modeled value based on the determination thatthe deviation exceeds the predetermined threshold.
 36. The one or morenon-transitory computer-readable media of claim 33, the operationsfurther comprising: determining that the deviation does not exceed thepredetermined threshold, wherein identifying the first modeled value ashaving the first level of impact on the model drift comprisesidentifying the first modeled value as being a secondary modeled valuein lieu of being a primary modeled value based on the determination thatthe deviation does not exceed the predetermined threshold.
 37. The oneor more non-transitory computer-readable media of claim 33, theoperations further comprising determining the model drift based ondeviations of each modeled value of the plurality of modeled values foreach factor of the plurality of factors from a set of predeterminedcriteria for the factor.
 38. The one or more non-transitorycomputer-readable media of claim 33, wherein causing the presentation ofthe third graphical representation comprises overlaying, on the seconduser interface, the third graphical representation of the extent towhich the first modeled value associated with the first selectablefeature contributes to the model drift.
 39. The one or morenon-transitory computer-readable media of claim 33, wherein presentingthe third graphical representation comprises presenting a pop-up windowdescribing predetermined criteria for a corresponding factor for thefirst modeled value.
 40. The one or more non-transitorycomputer-readable media of claim 33, wherein the first modeled value isa primary modeled value, and wherein the extent to which the firstmodeled value contributes to the model drift is determined relative toone or more other primary modeled values.